VANISHING OF QUASI-INVARIANT GENERALIZED 

FUNCTIONS 



DIHUA JIANG, BINYONG SUN, AND CHEN-BO ZHU* 

Abstract. Determination of quasi- invariant generalized functions 
is important for a variety of problems in representation theory, no- 
tably character theory and restriction problems. In this note, we 
review some new and easy-to-use techniques to show vanishing 
of quasi-invariant generalized functions, developed in the recent 
work of the authors (Uniqueness of Ginzburg-Rallis models: the 
Archimedean case, Trans. Amer. Math. Soc. 363, (2011), 2763- 
2802). The first two techniques involve geometric notions attached 
to submanifolds, which we call metrical properness and unipotent 
^-incompatibility. The third one is analytic in nature, and it arises 
from the first occurrence phenomenon in Howe correspondence. We 
also highlight how these techniques quickly lead to two well-known 
uniqueness results, on trilinear forms and Whittaker models. 



1. QUASI-INVARIANT GENERALIZED FUNCTIONS 

1.1. Generalized functions. Let M be a smooth manifold. Denote 
by C^{M) the space of compactly supported smooth functions on M, 
which is a complete locally convex topological vector space under the 
usual inductive smooth topology. Denote by D~°°{M) the strong dual 
of C^(M), whose members are called distributions on M. A distribu- 
tion on M is called a smooth density if under local coordinate, it is a 
multiple of a smooth function with the Lebesgue measure. Under the 
inductive smooth topology, the space D'^{M) of compactly supported 
smooth densities is again a complete locally convex topological vector 
space, which is (non-canonically) isomorphic to C^{M). Denote by 
(7~°°(M) the strong dual of D^{M), whose members are called gener- 
alized functions on M. By means of the integration pairing between 
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functions and densities, every smooth function (and more generally ev- 
ery locally integrable function) can be regarded as a generalized func- 
tion. The space C^(Af) of smooth functions in thus canonically and 
continuously embedded in C^°°(M), and it has a dense image. 

If : M ^ M' is a smooth map of smooth manifolds, then the 
pushing forward sends compactly supported distributions on M to com- 
pactly supported distributions on M' . If furthermore ^ is a submersion, 
then the pushing forward induces a continuous linear map 

0, : iy^{M) ^ iy^{M'). 

We define the puUing back 

(1) (j)* : C^°°(M') ^ Cr^{M) 

as the transpose of which extends the usual pulling back of smooth 
functions. The map (j)* is injective if is a surjcctivc submersion. 

Remark 1.1. Pulling hack is not canonically defined for distributions. 
For this reason, we work with generalized functions instead of distribu- 
tions. Informally speaking, "generalized functions transform like func- 
tions". 

1.2. Differential operators and transversality. For A; e Z, denote 
by DO(M)fe the Prechet space of differential operators on M of order 
at most k, which by convention is if A; < 0. It is well-known that 
every differential operator D : C^{M) C^[M) may be continuously 
extended to D : Cr°°(M) ^ Cr°°(M). 
We have the principal symbol map 

ak : DO(M)fc ^ r°^(M, S\T{M) ®m C)), 

where T(M) is the real tangent bundle of M, S'^ stands for the k-th 
symmetric power, and r°° stands for smooth sections. The continuous 
linear map is specified by the following rule: 

ak{XiX2---Xk){x)=Xi{x)X2ix)---Xk{x), and 

0'k\DO{M)k-i = 0) 

for all a; G M and all (smooth real) vector fields Xi, X2, ■ ■ • , Xj^ on M. 
Let Z he a. (locally closed) submanifold of M. Write 

Nz{M)^T{M)\z/T{Z) 

for the normal bundle of Z in M. Denote by 

(2) ak,z ■ DO(M)fe ^ r°°(Z, S'{Nz{M) ®m C)) 
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the map formed by composing 0"^ with the restriction map to Z, and 
followed by the quotient map 

T°°{Z, ^\T{M)\z ®R C)) ^ r°°(Z, ^\^z{M) ®M C)). 

Definition 1.2. (a) A vector field X on M is said to be tangential 
to Z if X{z) is in the tangent space Tz{Z) for all z E Z, and 
transversal to Z if X{z) ^ Tz{Z) for all z E Z; more generally 
(b) a differential operator D is said to be tangential to Z if for 
every point z E Z there is an open neighborhood Uz in M such 
that D\u^ is a finite sum of differential operators of the form 
If X 1X2 ■ ■ ■ Xr, where (f is a smooth function on f/^, r > 0, and 
Xi,X2, ■ ■ ■ ,Xr are vector fields on Uz which are tangential to 
Uz n Z. For D e DO(M)fe, it is said to be transversal to Z if 
<^k,z{D) does not vanish at any point of Z. 

We introduce some notations. For a locally closed subset Z of M, 
denote 

(3) cr^{M; z) = {fe cr^{u)\ supp(/) c z}, 

where U is any open subset of M containing Z as a closed subset. This 
definition is independent of U. For any differential operator D on M, 
denote 

(4) C-°°{M;Z;D) = {feCr^{M;Z)\Df = 0}. 

The following proposition is due to Shalika |Sh] . It asserts non- 
existence of certain generalized functions with support in Z. 

Proposition 1.3. Let Di be a differential operator on M of order 
k > 0, which is transversal to a submanifold Z of M. Let D2 be a 
differential operator on M which is tangential to Z . Then 

Cr^{M;Z;D^ + D2) = 0. 

Remark 1.4. Shalika uses the transversality technique to show that 
certain generalized functions cannot be supported in a Bruhat cell of 
strictly smaller dimension. See [Sh', Proposition 2.10]. This is a key 
step in his proof of uniqueness of Whittaker models in the Archimedean 
case. 

1.3. Invariant generalized functions and restriction to a slice. 

Let if be a Lie group, acting smoothly on a manifold M. Fix a char- 
acter X oil H. Denote by 

(5) q°°(M) = {/ G Cr^{M)\ f{hx) = xih)fix), for h e H} 

the space of x-equivariant generalized functions. 
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Let DJlhe a submanifold of M and denote 



PW. H xm-> M 

the action map. 

Definition 1.5. (a) We say that 97t is a local H slice of M if p<ffi 
is a submersion, and an H slice of M if pgjt is a surjective sub- 
mersion. 

(b) Given two submanifolds 3 C njt of M, we say that 3 is relatively 
H stable in M if 

mnH3 = ^. 

Note that the relative stable condition amounts to saying that -ff x 3 
is a union of fibres of the action map pot- The following lemma is 
elementary. 

Lemma 1.6. Let Wl be an H slice of M, and let "5 be a relatively H 
stable submanifold ofDJl. Then Z = is a submanifold of M, and 3 
is an H slice of Z . Furthermore if 3 is closed in OJl, then Z is closed 
in M. 

Now assume that 9Jl is a local H slice of M, and is a closed 
subgroup of H which leaves Wl stable. Let H act on H x Wl hj left 
multiplication on the first factor, and let H<x)x act on if x OJt by 

g{h, x) = {ghg~^, gx), g E H^, he H,x eDJl. 

Then the submersion pgjj is H intertwining as well as H^x intertwining. 
Therefore the pulling back yields a linear map 

: q~(M) ^ X an) n c^^{h x m), 

where X'm = xIh^- Schwartz Kernel Theorem [Scj and the fact 

that every invariant distribution on a Lie group is a scalar multiple of 
the Haar measure ( |Wa88t 8.A]), we have 

Consequently, 

q°°(ii X mt) n cr^iH x aji) = x ® cr^m- 

We thus have the following 

Proposition 1.7. There is a well-defined map which is called the re- 
striction to 9Jl.- 

by requiring that 

PmU) = X® fW- 
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The map is injective when DJt is an H slice. 



Remark 1.8. Aizenhud and Gourevitch [AG] have developed sophisti- 
cated techniques (called generalized Harish- Chandra descent) for work- 
ing with G-invariant generalized functions on a smooth affine G-variety, 
based on Luna's slice theorem |Lu] . 

2. Metrical properness and unipotent x-incompatibility 

2.1. Metrical properness. This notion requires that the manifold 
M is pseudo Riemannian, i.e., the tangent spaces are equipped with a 
smoothly varying family {{ , ) x '■ x & M} of nondegenerate symmetric 
bilinear forms. 

Definition 2.1. (a) A submanifold Z of a pseudo Riemannian man- 
ifold M is said to be metrically proper if for all z ^ Z, the 
tangent space Tz{Z) is contained in a proper nondegenerate 
subspace of T^(M). 
(b) A differential operator D G D0(M)2 is said to be of Laplacian 
type if for all x G M, the principal symbol 

Cr2(-D)(a;) = UiVi + U2V2 H h UmVm, 

where ui,U2, - ■ ■ , Mm is a basis of the tangent space Tx{M), and 
vi,V2, - ■ ■ ,Vm is the dual basis in T^i^M) with respect to ( , )x- 

Note that a Laplacian type differential operator is transversal to any 
metrically proper submanifold, from its very definition. Therefore the 
following is a special case of Proposition 11.31 

Proposition 2.2. Let Z be a metrically proper submanifold of M, and 
let D be a Laplacian type differential operator on M. Then 

Cr^{M]Z]D) = 0. 

Remark 2.3. The above proposition may be viewed as a form of uncer- 
tainty principle. The second and third named authors exploited metri- 
cal properness in their proof of Archimedean multiplicity one theorems 
[SZ2l Section 5]. 

2.2. Unipotent ^-incompatibility. As in Section 11.31 let H he a Lie 
group with a character x on it, acting smoothly on a manifold M. If 
a locally closed subset Z of M is if-stable, denote by C^°°{M; Z) the 
space of all / in C~°°{M] Z) which are x-equivariant. We shall use 
similar notations (such as C^°°{M; D) and C^°° {M ; Z ; D)) without 
further explanation. 
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Definition 2.4. An if -stable submanifold Z of M is said to be unipo- 
tently x-incompatible if for every zq G Z, there is a local H slice 3 of Z, 
containing Zq, and a smooth map (p : ^ H such that the foUowings 
hold for all 2; G 3: 

(a) 4){z)z = z, and 

(b) the linear map 

T,(M)/T,(Z) ^T,(M)/T,(Z) 

induced by the action of (f){z) on M is unipotent; 

(c) 7^ 1- 

The notion of unipotent x-incompatibility is of importance due to 
the following 

Proposition 2.5. Let Z be an H-stahle submanifold of M which is 
unipotently x-i'^compatible. Then C^°^{M] Z) = 0. 

Remark 2.6. Aizenbud, Ofjen and Sayag have proved an analog of the 
above proposition in an algebraic setting. See |AUSt Proposition 2.3]. 

2.3. A synthesis: M property. As before, let if be a Lie group 
acting smoothly on a manifold M, and let x be a character on H. We 
further assume that M is a pseudo Riemannian manifold. 

Definition 2.7. We say that an H -stable locally closed subset Z of M 
has M property if there is a finite filtration 

Z = ZoD Z,D ■■■D ZkD Zk+i = 

of Z by H -stable closed subsets of Z such that each Zi \ Zi^i is a sub- 
manifold of M which is either unipotently x-incompatible or metrically 
proper in M . 

As a combination of Propositions 12.21 and 12.51 we have 

Proposition 2.8. Let D be a differential operator on M of Laplacian 
type. Let Z be an H -stable closed subset of M having M property. 
Then 

Cr°°{M;Z;D) = 0. 

Remark 2.9. Subsets satisfying U^M property indeed arise in appli- 
cations. See |JSZl Sections 5 and 6], as an example. 

3. First occurrence in Howe correspondence 

First occurrence phenomenon in (local and global) theta correspon- 
dence was discovered by S.S. Kudla |Ku86j and S. Rallis |Ra] . In the 
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mid 1990's, Kudla and Rallis put forward their conservation conjec- 
tures on first occurrences in local theta (or Howe) correspondence, and 
their pioneering work have profound implications. We refer the reader 
to [KRj . The full conjectures are now proved, in a 2012 preprint by the 
second and third named authors |SZ3] . 

We shall only be concerned with the orthogonal-symplectic dual pair 
and the first occurrences of two very special characters. Note that in 
the dual pair setting, non-occurrence of characters amounts to van- 
ishing of quasi-invariant generalized functions. We shall focus on the 
Archimedean case. 

Let k = M or C. Let V be a (non-degenerate) quadratic space and 
W he a. symplectic space, over k, and consider the reductive dual pair 

(0(F),Sp(iy)) c Sp{V®W). 
Fix a nontrivial unitary character of k, and consider the smooth os- 
cillator representation ijJv,w associated to the dual pair (O(V^), Sp(l^)) 
and to the character ip. 

We fix a parity e G Z/2Z of dimV^. Write 

(6) 1 ^ {±1} ^ Sp^{W) Sp{W) 1 

for the unique topological central extension of the symplectic group 
Sp(iy) by {±1} such that it does not split if k is isomorphic to M, e is 
odd, and W is nonzero, and it splits otherwise. It is well-known that the 
oscillator representation ooy^w yields a representation of 0{V) x Sp^(iy). 

Denote by Itt{0{V)) the isomorphism classes of irreducible Casselman- 
Wallach representations of O(V^), and Irr(Sp^(H^)) the isomorphism 
classes of irreducible genuine Casselman-Wallach representations of 
Sp^iW). The reader may consult |Wa92l Chapter 11] for details about 
Casselman-Wallach representations. Throughout this section, vr de- 
notes a representation in Irr(0(V^)) and p denotes a representation in 
Irr(Sp^(Vr)). We are interested in occurrences of vr and p in the local 
theta correspondence |Ho89j . 

3.1. Orthogonal group. We first consider the case of orthogonal groups. 
Thus we fix V. Recall Kudla's persistence principle |Ku96j : if Wi, W2 
are two symplectic spaces, and diml^^i < dim 14^2, then 

Homo(y)(c<;v',vVi, 71") 7^ implies Homo(y) (wy^n^j ? ^) 7^ 0- 
Define the first occurrence index 

(7) n(7r) := min{-dimiy | Homo(y)(wy,H', vr) 7^ 0}. 
The following result is due to Przebinda |Przt Theorem C.7]. 
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Proposition 3.1. We have 

n(det) = dim V, 

where det stands for the determinant character of 0{V). 

Corollary 3.2. Let V be a finite dimensional non- degenerate quadratic 
space overk, and let the orthogonal group 0{V) act on V" diagonally, 
where n is a positive integer. If n < dim V , and if a tempered general- 
ized function f on V"^ is SO{V) -invariant, then f is 0{V) -invariant. 

3.2. Symplectic group. Now we consider the case of symplectic groups 
The results to be discussed are of a slightly different nature than the 
rest of this note, because the Sp^(H^) representation is not induced from 
a geometric action. Nevertheless, we shall present the results parallel 
to the case of orthogonal groups. 

We fix W. Kudla's persistence principle |Ku96j says that if Vi, V2 are 
two quadratic spaces belonging to the same Witt tower, and dim Vi < 
dimV2, then 

}lomsp^(w){(^v^,w,p) 7^ implies }lomsp^(w)i(^V2,w, p) 7^ 0. 

We also fix a Witt tower T of quadratic spaces. Define the first 
occurrence index 

(8) rarip) ■= min{diml^ \V eT, }iomsp^{w){^v,w, p) 0}. 

The following result is due to Loke |LLl Theorem 1.2.1], and it 
amounts to the determination of mT{C), where C stands for the unique 
one-dimensional genuine representation of Sp^(Vr) (when dim V is even). 
Recall that a quadratic space V is called quasi-split if its split rank 
> disii:^. Write dimW = 2n. 

Proposition 3.3. Assume that e is even. 

(i) // V is not quasi-split, then 

Homsp^(vt^)(a;y,H/,C) ^ 

implies that V has split rank > 2n, in particular dim V > 4n+4. 

(ii) // k = M and V is quasi-split and nonsplit, then 

Homsp^(i^)(wy,w,C) 7^ 
implies that V has split rank > n, in particular dim V > 2n + 2. 

Remark 3.4. The proof of the above proposition is through K-type 
computations. See [LLf Section 3]. It is worth mentioning that Propo- 
sitions W71\ and WT^ have their non- Archimedean analogs [Ral IKRj . and 
a uniform proof of these analogs is given in |SZ3t Section 5] . 
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4. Two QUICK APPLICATIONS 

As applications of the techniques discussed in this note, we present 
two quick and sweet consequences. See |JSZl Section 11]. 

4.1. Uniqueness of trilinear forms. Let k = M or C. Let H2 = 

GL2(k), and H2 be the following extended group: 

^2 = {l,r}xGL2(k), 
where the semidirect product is given by the action 

r{9) = g-'. 

Denote by X2 the character of H2 such that 

X2|GL2{k) = 1 and X2{,r) = -1. 

As in [JSZj . we use to denote (appropriate) space of tempered 
generalized functions. 

Lemma 4.1. Let H2 act on M2 = GL2(k) x GL2(k) by 
9ix,y) = i9xg'^,gyg'^), G GL2(k), 

and 

r{x,y) = {x\y'). 

Then 

cr/^m = 0. 

Proof. Using the same formula, we may extend the action of H2 on 
GL2(k) X GL2(k) to the larger space 0l2(k) x 0[2(k). It suffices to prove 
that 

C^HqW X = 0. 

Identify k with the center of 5t2(k). We have 

0l2(k) X gkik) = (skik) X s/2(k)) © (k x k) 

as a k linear representation of H2, where H2 acts on k x k trivially. 
Therefore it suffices to prove that 

q«(5Z2(k) X 5/2(k)) = 0. 

We view s[2(k) as a three dimensional quadratic space under the trace 
form. Under this identification, the action of H2 yields the diagonal 
action of 0(st2(k)) on sl2(k) x sl2(k), with X2 corresponding to the 
determinant character. So the required vanishing result is a special 
case of Corollary 13.21 □ 

The following theorem is proved in |Loj (in an exhaustive approach) , 
and its p-adic analog was proved much earlier in |Pral Theorem 1.1]. 



Theorem 4.2. Let V be an irreducible Casselman-Wallach represen- 
tation o/GL2(k) X GL2(k) x GL2(k). Then 

dimHomGL2(k)(^,C^2) < 1- 

Here we view GL2(k) as the diagonal subgroup o/GL2(k) x GL2(k) x 
GL2(k), X2 = Xkx°det is a character o/GL2(k), andxkx is an arbitrary 
character ofk^. 

Proof. By the Gehand-Kazhdan criterion |GKj (a general form is in 
|SZ1[ Theorem 2.3]), one just needs to show the following: let GL2(k) x 
GL2(k) act on 

G2,2,2 = GL2(k) X GL2(k) X GL2(k) 

by 

i9u 92)ix, y, z) = {gixgl, giyg{, gizgl), gi, g2 e GL2(k). 
Denote by X2,2 the character of GL2(k) x GL2(k) given by 

X2,2i9u 92) = Xkx idet{gi))xkx {det{g2)), gi, 5-2 e GL2(k). 

Then for all / G (^^2(^2,2, 2)) we have 

f{.x\y\z*) = f{x,y,z). 

To show the above, we observe that M2 = GL2(k) x GL2(k) x {I2} is a 
GL2(k) X GL2(k) slice of (^2,2,2, which is stable under H2 = {{x,x^^) \ 
X G GL2(k)} C GL2(k) x GL2(k) and r. The result then follows from 
Lemma 14.11 in view of Proposition 11.71 □ 

4.2. Uniqueness of Whittaker models. Let G be a quasisplit con- 
nected reductive algebraic group defined over M. Let B be a Borel 
subgroup of G, with unipotent radical N. Let 

XN : N(R) ^ 

be a generic unitary character. The meaning of "generic" will be ex- 
plained later in the proof. 

The following theorem is fundamental and well-known. For G = 
GL„, this is a celebrated result of Shalika |Sh] . A proof in general may 
be found in |CH]VH Theorem 9.2]. We shall give a short proof based on 
the notion of unipotent x-incompatibility. 

Theorem 4.3. Let V be an irreducible Casselman-Wallach represen- 
tation of G{M.) . Then 

dimHomN(iR)(V,CxN) < 1- 
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Proof. Define a Casselman-Wallach distributional representation to be 
the strong dual of a Casselman-Wallach representation. The current 
theorem can then be reformulated as follows: the space 

jjx^' = {^^ G I = Xn (c/)m for all g G N(M)} 

is at most one dimensional for every irreducible Casselman-Wallach 
distributional representation U of G(M). 

Let B be a Borel subgroup opposite to B, with unipotent radical N. 
Then T = B fl B is a maximal torus. Let 

XT ■■ T(M) ^ 

be an arbitrary character. Then 

U{xt) ={f e C-°°(G(M)) I fitnx) = XT{t)f{x) 

for all t e T{R),ne N(M)} 

is the distributional version of nonunitary principal series representa- 
tions. By Casselman's subrepresentation theorem (in the category of 
Casselman-Wallach distributional representations), it suffices to show 
that 

(9) dimU{xT)^^ < 1, for any Xt- 
Let 

Hg = B(M) X N(R), 

which acts on G(R) by 

(b, n)x = bxn~^. 

Write 

Xcitn^n) = XT(t)XN(n), 
which defines a character of Hg- Then (Q is equivalent to 

(10) dimC;^;f(G(M)) < 1. 

Let W be the Weyl group of G(]R) with respect to T. We have the 
Bruhat decomposition 

G(M) = y G^, with G^ = B(M)w;N(M). 

From this we form a Hg stable filtration 

= G° c G^ c G^ c ■ ■ ■ c G" = G(M) 

of G(]R) by open subsets, with G"*^ = B(M)N(R) and every difference 
G* \ G*-i a Bruhat cell G^, for i > 2. 

Clearly (JTOl) is implied by the following two assertions: 

(11) dimq^(Gi) = l; 
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and 

(12) if / G C^^iG') vanishes on G'-\ then / = 0, 

for i > 2. The equahty is clear as = B(M)N(M). For ([I2]), we 
write 

Qt ^ Q«-i _ with w a non-identity element of W. 

The genericity means that xn has nontrivial restriction to N(]R) fl 
m;-1(N(M))u;. Pick 

n = w'^^nw e N(R) r]w~\N(R))w 

so that Xn('t-) 7^ 1- Then {n,n) G Hq satisfies 

(n, r;,)ty = w, and Xg(^, 'T-) = Xn(^) 7^ 1- 

Consequently, is unipotently Xc-incompatible. Now (IT^ follows 
from Proposition 12.51 □ 
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